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abstract: This paper discusses a pilot study 
of the effects of a curriculum focus intervention 
on students' prealgebra achievement. Elements 
of the intervention include identification of 
high-priority learning objective; structured re- 
peatable testing; and a coherent , rubric-based 
feedback component. This research differs from 
traditional mastery learning research in that 
it focuses on a subset of high-priority learning 
objectives, as opposed to the entire curricu- 
lum, and focuses on assessing students' abil- 
ity to structure, represent, and communicate 
their processes and thinking skills, as opposed 
to assessing only whether the solution and pro- 
cess are correct. Students in the treatment and 
control groups were given general (not math- 
ematics specific) academic efficacy measures, 
a course-specific efficacy measure, and a com- 
mon course final exam. Only the differences in 
the means on course-specific measures were sta- 
tistically significant, with the treatment group 
outperforming the control group on both the 
course-specific efficacy measure and the final. A 
possible negative effect was that students in the 
treatment group dropped out at a higher rate 
than students in the control group. 

Though the phrase a “mile wide and an inch 
deep” was coined to describe school mathemat- 
ics (Schmidt, McKnight, & Raizen, 1997), it is 
also very descriptive of manyist-year college 
courses. Postsecondary developmental algebra 
sequences— pre or introductory algebra and in- 
termediate algebra — are often dense with topics, 
push students at a breath-taking pace, and cover 
many of the same topics in both courses, as if 
closure is neither expected nor achieved. 

This curriculum issue was noted by Stein- 
fort (1996) who reported on a collaborative 
effort by 14 Michigan community colleges to 
reform their developmental algebra curricu- 
lums. Steinfort wrote that one of the problems 
motivating the reform was that “as mathematics 
educators, we were racing through material with 
lack of realistic applications, and there was too 
much course content for effective conceptual 
understanding. In addition, many of us were 
re-teaching Elementary Algebra topics in Inter- 
mediate Algebra” (p. 2). More recently, Epper 


and Baker (2009) criticized current best-selling 
community college developmental mathematics 
textbooks and software packages, asserting they 
replicate K-12 curriculums by presenting long 
lists of seemingly unrelated topics. 

Yet textbooks and software packages that are 
packed with topics do not appear to reflect the 
curriculum desires of college developmental 
mathematics teachers. As reported in the ACT 
National Curriculum Survey 2005-2006 (ACT, 
2007), which asked teachers what they find to 
be important about curriculum, “postsecondary 
mathematics teachers of entry-level and remedi- 
al courses agree in favoring ‘depth and rigorous 
understanding of fundamental skills, whereas 
high school mathematics instructors more high- 
ly value ‘breadth” (ACT, p. 20). In this study, re- 
medial mathematics teachers “consistently rated 
understanding of fundamental mathematics as 
more important than exposure to more esoteric 
mathematics content topics for success in their 
courses” (ACT, p. 19). However, the ACT survey 
fell short of identifying exactly which topics are 
important enough to be called fundamental by 
developmental mathematics teachers. As Epper 
and Baker (2009) point out, “while there is broad 
agreement on the importance of both computa- 
tion fluency and conceptual understanding, the 
issue of which skill should be taught, and in 
what order, has not been resolved” (p. 4). 

Thus, there is a need for research that identi- 
fies the most important concepts and skills for 
each course in the developmental algebra se- 
quence and that also examines the achievement 
effects for students who master those topics. 
This article serves as a starting point for such 
research by reporting a pilot study which shows 
that when a small subset of high priority learn- 
ing objectives for a prealgebra course is identi- 
fied and agreed upon by the faculty at an institu- 
tion, it is reasonable to expect students to master 
these objectives completely (in terms skill profi- 
ciency and standards -based proficiency such as 
students’ abilities to structure and communicate 
their responses), not simply come close. It is also 
shown that students who do achieve mastery on 
these high-priority learning objectives display a 
measureable positive effect in their mathemati- 
cal achievement as measured by the courses 
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common final exam and their mathematics self- 
efficacy beliefs. 

This research differs from traditional mastery 
learning research in that it focuses on a subset 
of high priority learning objectives, as opposed 
to the entire curriculum, and focuses on as- 
sessing students’ ability to structure, represent, 
and communicate their processes and thinking 
skills, as opposed to assessing only whether the 
solution and process are correct. 

Mastery Learning and 
Feedback Research 

The concept of mastery learning is not new to 
education. The 1920s and 1930s saw several sys- 
tems of instruction requiring mastery on formal 
assessments before students were moved to new 
material (Bloom, 1968; Kulik, Kulik, & Bangert- 
Drowns, 1990). Possibly best known today is the 
work of Benjamin Bloom from the late 1960s 
and early 1970s, which he called Learning for 
Mastery (LFM). Motivated by his classroom 
observations that teachers rarely varied their 
instruction practices to fi t individual learn- 
ers’ needs, Bloom devised a mastery process in 
which the curriculum is divided into units that 
are then taught, formatively assessed, and, if 
needed, retaught and reassessed (Gusky, 2005). 

Required in Bloom’s model is that this first 
assessment be followed by immediate “individu- 
alized” corrective instruction for those students 
who need it and enrichment activities for those 
students who don’t. Once students receive cor- 
rective instruction, they are then reassessed pri- 
or to the entire class moving on to the next in- 
structional unit. Over the years, several studies 
have linked LFM to positive effects on student 
achievement (Kulik et al., 1990). 

Also associated with the term “mastery” is 
Keller’s Personalized System of Instruction (PSI; 
Keller, 1968). PSI is similar to LFM in that the 
course materials are divided into units to be 
mastered before moving on to the next unit. 
PSI differs from LFM in that lessons are largely 
presented through written materials, which 
students move through at their own rates. PSI 
students who fail unit quizzes must retake quiz- 
zes until they are able to demonstrate mastery. 
PSI has also been shown to increase student 
achievement (Kulik et al., 1990). 

Criticisms of LFM and PSI include that LFM 
requires users to design a supplemental curricu- 
lum for learners who demonstrate mastery on 
the first attempt and that PSI programs may al- 
low learners to continue struggling at their own 
pace semesters after other students have com- 
pleted the entire course. The latter is in contrast 
to the Kulik et al. (1990, p. 286) finding that the 
largest effect sizes occurred in mastery programs 


that required students to move through course 
materials at the teacher’s pace, not the student’s 
individual rate. Another criticism is that stu- 
dents who experience PSI or LFM instruction 
appear to drop out at a higher rate when com- 
pared to students who experience conventional 
instruction (Kulik et al.). 

One aspect of the programs that has not been 
criticized is their use of feedback. Both LFM 
and PSI placed heavy reliance on a feedback 
loop, arguably the most prominent feature of 
these programs. In their original discourse, both 
Bloom and Keller expounded emphatically on 
the importance of feedback to their programs. 
Bloom (1968), for example, emphasized that the 
feedback loop between an excellent tutor and his 
or her students was what he sought to model. 
Similarly, Keller (1968) went to great lengths 
to downplay lecture and glorify the postassess- 
ment feedback process between a “proctor” and 
student. Moreover, in their meta- analysis of 
mastery programs, Kulik et al. (1990) reported 


Feedback can be one of 
the most powerful single 
modifications to classroom 
practices for increasing 
student achievement. 


that the strongest positive effect sizes come from 
mastery studies that used control subjects who 
received less feedback on quizzes than treatment 
subjects. They further reported that equating the 
amount of feedback between the control and 
treatment groups reduced the “size of the effect.” 
Couple this finding with the finding of Marzano, 
Pickering, and Pollack (2001) -that feedback can 
be one of the most powerful single modifica- 
tions to classroom practices for increasing stu- 
dent achievement in its own right- and one can- 
not help but wonder what portion of the positive 
effects from mastery learning can be attributed 
to the feedback loop alone. 

When the feedback component is eliminated 
from a mastery program, the program is more 
commonly called repeatable testing. Mixed re- 
sults have been reported for repeatable testing. 
Juhler, Rech, From, and Brogan (1998) found 
that repeatable testing increased scores on the 
test being repeated but found no increase on 
students’ fi nal exam scores. Fehlen (1976) on 
the other hand found strong positive results for 
repeatable testing. In his study, three treatment 
groups were used, two of which were given the 
option of repeating assessments. Of these two 
treatment groups, one included mandatory tu- 


toring. Fehlen contended that there was little 
difference between the effects for the tutoring 
and nontutoring groups, and he argued that re- 
peatable testing was the factor that accounted 
for the positive gains. In contrast, Dunkelberger 
and Heikkinen (1984) found no supporting 
evidence that repeatable testing “by itself” con- 
tributed significantly to student achievement. 
Dunkelberger and Heikkinen did mention that 
tutoring was available to treatment students but 
gave no indication as to whether it was utilized. 

Neither Fehlen nor Dunkelberger and Heik- 
kinen discuss feedback in detail, and therefore 
it cannot be assumed that the term “tutoring” 
includes a well-defined feedback component. 
Since both Bloom and Keller use the term “tu- 
toring” for contexts in which feedback on stu- 
dents’ work is a critical component, and Fehlen 
and Dunkelberger and Heikkinen do not ex- 
pound on the meaning of the term, the Fehlen 
and Dunkelberger fi ndings shed little light on 
the importance of feedback in the mastery pro- 
cess. 

Although it may be difficult to isolate the ef- 
fect of feedback in the mastery learning process, 
much is known about how to maximize feed- 
back’s effect. According to Marzano et al. (2001), 
the strongest effects come from feedback that 
is corrective, timely, and criterion-referenced 
(i.e., feedback that indicates the specific “level” 
of the knowledge). Criterion-based feedback is 
distinct from norm-referenced feedback, which 
informs students where they stand in relation- 
ship to other students, in that it focuses on spe- 
cific, well defined learning objectives. To achieve 
criterion-referenced feedback, Marzano et al. 
suggest using a scoring rubric. 

The power of a scoring rubric appears to 
go beyond its use as a vehicle for feedback. It 
also better equips educators in communicat- 
ing achievement standards to students prior to 
the assessment. Schafer, Swanson, Bene, and 
Newberry (1999) have found that students of 
teachers trained in their 4 -point scoring rubric 
had statistically significant higher achievement 
scores on both selected response items (multiple 
choice) and constructed response items (open 
response), suggesting that knowledge of the ru- 
bric produced a learning advantage that general- 
ized across assessment formats. 

The Focus Intervention Compared 
to Mastery Learning 

Because Bloom (1968), Keller (1963), and Ku- 
lik et al. (1990) have stressed the importance 
of coupling a feedback process with repeatable 
testing, the intervention described in this article 
(the Focus Intervention) incorporates a well- 
structured feedback loop after each assessment. 
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Yet, despite the use of repeatable testing and a 
formative assessment/feedback loop, the Focus 
Intervention is not “mastery learning” in the 
canonical sense. Th e biggest difference is that 
in the Focus Intervention the curriculum is not 
subdivided into units of content to be mastered 
before moving onto the next unit of content. The 
researchers in this study do not believe that all 
the content in a mathematics curriculum is of 
equal weight, nor do we believe that all content 
should be mastered immediately. Marzano et al. 
(2001), for example, have summarized research 
that suggests it takes 25 repetitions to achieve 
70% proficiency on new skills and upward of 
1000 repetitions for 100%. 

Further, the question of when and what top- 
ics should be mastered is particularly problem- 
atic in mathematics. In mathematics, many top- 
ics are foundational and are utilized continually 
throughout the course. Th erefore, the repeti- 
tion needed often occurs within other contexts. 
Thus, the Focus Intervention considers the 
curriculum as a whole and identifies subsets of 
content deemed of very high priority (possibly 
even among the most important concepts, pro- 
cedures, and problem strategies to be taught in 
the course) by a panel of experts. Further, the 
intervention focuses not only on the knowledge 
needed to solve the problems but also on the 
students abilities to communicate that knowl- 
edge using the language of mathematics, an im- 
portant objective of any prealgebra course. 

Another difference between the Focus In- 
tervention and “mastery learning,” as originally 
proposed in LFM and PSI, is the requirement 
that students achieve 100% mastery on the items 
to receive credit for the assessment. Bloom 
(1968) has never defined a level of mastery in 
terms of a percent. Instead, he targeted moving 
90% of the students to the level of mastery typi- 
cally obtained by the top 5% of students. Keller 
(1968), as mentioned earlier, has relied heavily 
on the judgment of the proctor to determine if 
a student had mastered the content. More re- 
cent researchers have defined mastery levels in 
a rather ad hoc way. Kulik et al. (1990, p. 268) 
report that some studies used percent correct 
criteria as low as 56% . However, they 
also report that the “especially large” 
effect sizes are associated with studies 
defining the mastery criteria as 100% 
correct. 

Thus, the Focus Intervention prom- 
ises to examine a gap in the literature 
beyond the question of whether pre- 
algebra students can be expected to 
master learning objectives completely. 
Specifically it examines the question: 

Is it also possible to produce positive 
effects on student performance with a 


focused mastery program that includes a strong 
feedback component within the constraints of 
conventional instruction that does not divide 
the curriculum into units assumed to be of equal 
importance? 

Research Questions 

To clarify, the research questions are restated as 
follows: 

• If a small subset of high-priority learning ob- 
jectives for a prealgebra course is identified 
and agreed upon by the faculty at an insti- 
tution, is it reasonable to expect students to 
master these objectives completely (in terms 
of skill proficiency and standards-based pro- 
ficiency such as students’ abilities to struc- 
ture and communicate their responses), not 
simply come close? 

• Will students who achieve mastery on these 
high-priority learning objectives display a 
positive measureable effect in their math- 
ematical achievement as measured by the 
courses common final exam? 

• Will students who do achieve mastery on 
these high priority learning objectives dis- 
play a positive measureable effect in their 
academic beliefs and strategies? 

Method 

Site and Participants 

The study was conducted at a doctoral-granting 
public university in a western state. The classes 
were taught by full-time faculty who exclusively 
taught developmental students at an extension of 
a 2 -year campus. The prealgebra course studied 
was the lowest level of mathematics offered; stu- 
dents were placed into the course with standard- 
ized mathematics test scores lower than 21 ACT, 
490 SAT, or the equivalent on a department- de- 
veloped placement test. Although the vast ma- 
jority of students in the study were enrolled at 
the 4 -year campus (92%) the course was desig- 
nated as below college level. Twenty-two percent 
of the students in the study were admitted to the 
university provisionally, meaning that they did 


not meet the minimum requirements for admis- 
sion, were restricted to 14- credit hours, and were 
required to take developmental courses in math- 
ematics and English in their 1 st semester. Th e 
mean number of credit hours attempted by the 
students during the study was 12.4 ± 3.6 (mean 
± SD) with 86% of the students designated full- 
time (more than 7 credits). Students were split 
closely with respect to gender (53% male). The 
group was predominately White (86%), with 8% 
Black, 3% Hispanic, and 3% not reporting. The 
mean student age was 22.4 ± 6.1 years. Student 
ages ranged from 17- 50, with 22% of the stu- 
dents nontraditional in age (over 24-years old). 
Students who did not take the final exam were 
excluded from the data and discussion. 

Control and Treatment Group Selection 

In the Spring 2008 semester, treatment and 
control college-level prealgebra classes were 
randomly selected from two instructors’ course 
assignments. One instructor (the second au- 
thor) was assigned three sections, and two of 
his courses were randomly selected as treatment 
(Focus Intervention). The other instructor was 
assigned two sections, and one of her classes 
was randomly selected as treatment. Th e two 
remaining sections were used as control groups. 
These courses had very small enrollment. Th e 
treatment group contained 21 students, and the 
control group contained 15. Table 1 illustrates 
this breakdown by instructor. 

The Focus Intervention Experimental 
Procedure 

A focal-point test (FPT; see Appendix A), was 
developed by the two prealgebra teachers and 
then evaluated for content and appropriate- 
ness by the other fi ve faculty teaching subse- 
quent courses. The first focal-point assessment 
was administered a few days after the midterm 
point in the course. Three parallel versions were 
made available for students who did not receive 
100% mastery on the assessment. No changes to 
the curriculum or structure of the course were 
needed to accommodate the Focus Intervention. 
Students in both the treatment and the control 
groups were given practice versions of 
the focal-point assessment. Students 
in the treatment group were given 
the rubric (see Appendix B) and told 
that they must obtain a score of 100% 
based on the rubric or they would be 
assigned a 0% . Students who did not 
receive 100% were required to meet 
with the instructor one-on-one dur- 
ing office hours to get individualized 
feedback before being allowed to 
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Table 1 

Courses in the Study 


Section 

Number 
of Students 

Treatment 
or Control 

Instructor 

01 

10 

Control 

A 

02 

10 

Treatment 

A 

04 

7 

Treatment 

B 

05 

5 

Control 

B 
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Table 2 

Factor Analysis Summary for Items on the MSES2 

Factor Items and Loadings Proportion of 

Variance Explained 


1 

Item 1-.499 

(Working with 

Item 2-.653 

different number 

Item 3-. 917 

representations) 

Item 6-. 736 

2 

(Working with 

Item 4-.678 

integers and 

Item 5-. 860 

equations) 

Item 7-. 578 
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retake the FPT. Students in the control group 
were not told they needed to master the mate- 
rial. They were told instead that the assessment 
would count heavily towards their grade. 

The focal-point assessment was then given to 
both the treatment and the control groups as an 
in-class test. The control groups work was graded 
by the instructor in a manner consistent with the 
other tests in the course. To address concerns that 
the treatment group simply had more exposure 
to the problems on the assessment, the control 
group was allowed to rework the missed prob- 
lems outside of class for half of the missed points 
back. (It should be noted that a few students in 
the control group chose not to rework the assign- 
ment, despite the fact that both the instructors 
and a free tutor were available to help students 
rework problems and that, in some cases, these 
points would have significantly improved the stu- 
dent’s standing in the course.) 

For students in the treatment group who did 
not master the assessment on the first try, in- 
dividual feedback meetings were held with the 
instructor. These meetings consisted of a discus- 
sion with the student regarding how his or her 
work and communication was measured with 
respect to the rubric. After the meeting, students 
could then schedule a retake of the problems not 
yet mastered with a proctor outside of class (re- 
peatable testing). The process would repeat un- 
til the student mastered all of the problems or 
had taken the assessment four times. Students 
were told that if they did not receive 100% by the 
fourth assessment, they would receive a o for the 
assessment. 

The decision to allow students to take the 
assessment only four times was made for both 
practical and pedagogical reasons. Pedagogical- 
ly, it was believed that more than one attempt at 
the assessment was needed for the feedback loop 
to be effective. However, a limited number of at- 
tempts was also needed to encourage students 
to make every effort to pass the assessment on 
every attempt. Also, the authors had done sin- 
gle-course pilots (without the self-efficacy mea- 
sures) and found that a few students were un- 
able to pass the assessment on a third attempt. 
This suggested more than three attempts might 
be needed. Practical considerations, such as the 
amount of time an instructor had to devote to 
the grading and feedback process, influenced 
the decision to allow only four attempts at the 
assessment. 

The unit of analysis was the students. Com- 
parison of the treatment and control was per- 
formed using available SAT and ACT data and 
scores from the courses fi rst four unit assess- 
ments, which were the same exams for all sec- 


tions. SAT mathematics data 
was available for five treatment 
students and four control stu- 
dents, with means and standard 
deviations of 422 (SD= 54.5) 
and 417.5 (££>=69.46). ACT 
mathematics data was avail- 
able for 10 treatment and 9 
control students, with means 
and standard deviations of 16.9 
(SD= 1.72) and 16.3 ( SD=i.y 3). 

Twenty- six percent of the stu- 
dents did not submit either 
ACT or SAT data (enrollment 
into this course, as the lowest 
level mathematics course, did 
not require placement data). 

The most consistent com- 
parison data were the students’ performance on 
the first four course unit assessments. The treat- 
ment group mean was 90.87 (£0=0562) and the 
control group mean was 89.33 (^£>=0744). A t- 
test did not reject the null hypothesis that these 
group means were equal ( £=0.71 , ^=.4847), so 
one could assume group means were equal. 

Measures 

Statistical analysis of the three following mea- 
sures was performed using R, a language and 
environment for statistical computing, (R De- 
velopment Core Team, 2007) and the R pack- 
age ltm (Rizopoulos, 2006). In many cases, the 
sample sizes needed for robust statistical analy- 
sis were not available in this study. In such cases, 
the statistical results are reported with the limi- 
tations disclosed. Despite the limitations, the 
reported analysis is likely to be useful to future 
researchers or practitioners in designing similar 
studies. 

Mathematics Self-Efficacy (MSES2). The 
instrument in Figure 1 was modeled after the 
Mathematics Self-Efficacy Scale (MSES; Nielsen 
& Moore, 2003). Shown to be valid and reliable 
for students whose ages range from 14 to 16, the 
MSES involves stu- 
dent self-assessment 
of competence in rela- 
tion to specific math- 
ematics tasks. Because 
the curriculum en- 
countered by students 
in this age range has 
many similarities to 
curriculum encoun- 
tered in the preal- 
gebra course in this 
study (topics include 
operations on frac- 
tions, decimals, ratios, 
proportions, percents, 


signed numbers, and one-variable equations), the 
MSES was believed to be a good model for the 
development of instrument for this study’s adult 
audience. 

The seven questions on the MSES2 (see Fig- 
ure 1) resulted from adapting questions from the 
MSES to this study’s college audience and were 
ordered randomly. The survey was given at the 
beginning of the semester (pretreatment) and at 
the end (posttreatment) to participants in both 
the treatment and control groups. Results from 
the 36 students’ pretreatment surveys were used 
for a reliability and validity check. Construct va- 
lidity was checked using factor analysis. 

For factor analysis, Bryant and Yarnold (1995) 
assert that the ratio of subjects to items should 
be no lower than that five, and this sample meets 
that criterion. However, Bryant and Yarnold 
also endorse a “rule of 200,” which asserts that 
sample sizes should include at least 200 subjects. 
This criterion is not met by the sample. 

The survey showed strong internal reliabil- 
ity (Cronbach’s alpha=.8o2), and factor analysis 
did not reject the null hypothesis that the sur- 
vey contained 2 factors: working with different 
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The following questions ask you to estimate you own mathematics ability. 
On a scale of 1 (not at all confident) to 5 (very confident), how confident 
are you that you can perform each of the following mathematics tasks in 
a mathematics test? 

1 . Work with fractions. 

2. Work with percents. 

3. Work a problem using proportions. 

4. Work with whole numbers. 

5. Work with negative numbers. 

6. Work with decimals. 

7. Solve an algebraic equation. 

Figure 1. Questions developed for the MSES2 using the 
Mathematics Self-Efficacy Scale as a model. 
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number representations and working with inte- 
gers and equations. The only surprising finding 
was that item 7 , working with equations, cor- 
related with a factor containing items 4 and 5 , 
working with whole and negative numbers. A 
possible explanation is that the algebraic equa- 
tions solved in this course were linear and the 
majority of these equations involved integer co- 
efficients (whole numbers and negative whole 
numbers). Further, the procedures involved in 
solving these equations — such as order of oper- 
ations and use of distributive properties — very 
closely mimic procedures for simplifying integer 
expression, a prerequisite for solving equations. 
Thus, the difficulty students have manipulating 
integer expressions is likely carried over to solv- 
ing algebraic equations containing these expres- 
sions. Table 2 summarizes the results. 

The survey also showed predictive validity. 
The treatment and control groups’ posttreat- 
ment MSES 2 survey was moderately correlated 
with the students’ fi nal exam scores (r=. 489 , 
p=. 0025 ), explaining roughly 25 % of the vari- 
ance on the final exam. 

General Academic Beliefs (GAB). The sec- 
ond survey (see excerpts in Figure 2 ) was de- 
signed to measure more general self-efficacy, in- 
trinsic value, test anxiety, cognitive strategy use, 
and self-regulation. Th e survey was created by 
selecting items from the Motivated Strategies for 


(Rank 1 to 7. l=not at all true for me. 7=very 
true for me.) 

Motivational Beliefs: 

A. Self-Efficacy 

5. I’m certain I can understand the ideas 
taught in this course. 

B. Intrinsic Value 

4. I think that what we are learning in 
this class is interesting. 

C. Test Anxiety 

3. I am so nervous during a test that I 
cannot remember facts I have 
learned. (Reverse) 

D. Cognitive Strategy Use 

17. When learning new math I try to 
connect the new ideas with math I 
already know. 

E. Self- Regulation 

24. I ask myself questions to make sure I 
know the material I have been studying. 

Figure 2 . Selected questions from 
the General Academic Beliefs Survey 
developed using Motivated Strategies 
for Learning Questionnaire as a model. 


Learning Question- 
naire survey. Pintrich 
and DeGroot ( 1990 ), 
who examined the 
survey’s reliability, va- 
lidity, and categorical 
correlations to aca- 
demic performance 
for seventh-grade sci- 
ence and English stu- 
dents. Items were ordered randomly and num- 
bered as in Figure 2 . The survey was given at the 
beginning of the semester (pretreatment) and at 
the end (posttreatment) to participants in both 
the treatment and control groups. Results from 
the 36 students’ pretreatment surveys were used 
for a reliability and validity check. The ratio of 
subjects-to -variables did not meet the “rule of 
200 ” endorsed by Bryant and Yarnold ( 1995 ) 
and, therefore, validity inferences should be 
noted with caution. 

The survey showed strong internal reliability 
(Cronbach’s alpha=. 722 ), and factor analysis did 
not reject the null hypothesis that the survey con- 
tained 4 factors. The first factor contained three 
items from each of the categories intrinsic value, 
cognitive strategy use, and self-regulation (B, D, 
E) for a total of nine items. Factor 2 contained 
all of the items from self-efficacy (A) and factor 
3 contained all four items from text anxiety (C). 
The fourth factor only explained approximately 
5 % of the survey data’s total variation and was 
not viewed as important. Test anxiety and self- 
efficacy do appear to form their own factors or 
constructs whereas the remaining items appear 
to form a separate construct. The fact that three 
categories correlated with the same factor is not 
surprising since Pintrich and DeGroot ( 1990 ) 
found these variables highly correlated (p<.ooi). 

Course fi nal exam. A comprehensive fi nal 
was given at the end of the semester. This final 
contained 35 items 
designed and selected 
by a team of five ex- 
perienced instructors 
of the course (face va- 
lidity). Th e fi nal was 
multiple choice and 
items were either cor- 
rect or not. Old final 
exam data was com- 
bined with final exam 
data from this study to 
determine its validity 
and reliability, 71 = 117 . 

The fi nal showed 
moderate internal re- 
liability (Chronbach’s 
alpha=. 628 ). Princi- 
pal component analy- 


Table3 

MANOVA on Final Exam Scores and MSME2/GAB Scores 



df 

Pillai’s Trace 

approx F 

df 

Pr(>F) 

Treatment 






Versus Control 

1 

0.40096 

2.25899 

8 

0.0542 

Residuals 

34 






sis was used to explore components within the 
assessment. Using the Kaiser Criteria (drop all 
components with eigenvalue less than one), no 
components were identified. The fact that the fi- 
nal exam score for students in the study showed 
moderate correlation (r=. 489 ) with the MSES 2 
data demonstrates some convergent validity of 
the final. 

Results 

Data Analysis 

Multivariate Analysis of Variance (MANOVA) 
was performed with the independent variable 
treatment ( 1 ) or control (o) and dependent vari- 
ables of the final exam score and the gains (post- 
pre) on the MSME 2 and the GAB (parts A, B, 
C, D, E, B+D+E each representing their own de- 
pendent variable). The null hypothesis was that 
the mean scores on each of the measures (the 
dependent variables) were the same for both 
groups, treatment and control: 

p ^ , where represents the 
mean score for the i th mea- 
H 0 : = ^. 2,r sure (ordered as listed above) 

for the j th group, treatment 
,t) (t) or control (c). This analy- 
sis was followed by f-tests to determine which 
variables represented statistically significant 
differences between treatment and control. 
Correlation coefficients were examined for all 


"aO 

A h, c 

f 

vA h*) 
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Table 4 

t-7esf Results for Individual Measures 


Measure 

Treatment (n 

=21) 

Control ( n= 

=16) 

f(p-v alue) 

Mean 

Gain/Score 

SD 

Mean 

Gain/Score 

SD 

GAB Part A 

-0.76 

6.36 

-2.27 

5.34 

0.75 (.4601) 

GAB Part B 

0.05 

4.33 

-1.33 

2.02 

1.15 (.2600) 

GAB Part C 

2.05 

5.85 

0.07 

4.93 

1.07 (.2927) 

GAB Part D 

-6.81 

4.89 

-9.80 

4.89 

1.09 (.2843) 

GAB Part E 

- 2.00 

8.25 

-2.60 

3.07 

0.27 (.7903) 

GAB Parts B,D,E -8.76 

21.49 

-13.73 

8.23 

0.85 (.4015) 

MSES2 

6.57 

5.69 

2.33 

4.47 

2.40 (.0219*) 

Final Exam 

0.9076 

0.0837 

0.8527 

0.0708 

2.13 (.0406*) 


* Denotes difference significant at the .05 significance level (two-tailed) 
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posttreatment measures. Statistical analysis was 
performed using R language and environment 
for statistical computing (R Development Core 
Team, 2007). Table 3 summarizes the results of 
the MANOVA. Overall, the differences in means 
of the dependent variables are significant at the 
10% level (^=0.0542). 

Table 4 summarizes the results of t-tests per- 
formed on individual measures. GAB measures 
intrinsic value, cognitive strategy use, and self- 
regulation ratings were combined for one rat- 
ing (based on the factor analysis). The mean of 
this rating declined for both the treatment and 
control group. Although the mean decline was 
smaller for the treatment group than the control 
group, hypothesis testing suggested the differ- 
ence in the observed means was due to chance. 
A similar result was found for the self- efficacy 
measure. Th e mean ratings for test anxiety 
(items reversed in Table 4 ) also decreased for 
both groups. The mean decrease was larger for 
the treatment group but hypothesis testing sug- 
gested the difference in the observed means was 
also due to chance. 

The fi ndings for the generic instrument 
(GAB) are interesting when compared with the 
subject- specific instruments (MSES2 and the 
final exam). Both of these subject- specific mea- 
sures showed that the treatment group outper- 
formed the control, and the results are statisti- 
cally significant. Also interesting is the fact that 
both treatment and control had positive gains on 
the subject specific self-efficacy instrument but 
negative gains on the generic self-efficacy instru- 
ment. 

Table 5 (p. 34) shows the correlation matrix 
for all posttreatment measures. All 36 students 
(treatment and control) are included in the anal- 
ysis. Note that all of the measures from the GAP 
correlate, and this correlation is statistically sig- 
nificant. Th is fi nding is consistent with that of 
Pintrich and Groot (1990). Table 5 also shows 
only MSES2 as a significant predictor of the final 
exam and that this measure exhibits statistically 
significant correlation only with the test anxiety 
measure from the GAB. Th is fi nding suggests 
that test anxiety is related to subject-specific self- 
efficacy beliefs. In summary, the larger mean 
gains, treatment group over the control group, 
were statistically significant only on course-spe- 
cific measures: the MSES2 and final exam. 

Retention Data 

The effect of the Focal Point Intervention on 
the retention of students may be hard to as- 
sess by this study. Raw data indicate that re- 
tention (percent of students on the 15th day of 
class who complete the course) in the treatment 

CONTINUED ON PAGE 34 
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class (64%) is lower than in the control (73%), 
which would be consistent with other mastery 
programs. However, it should be noted that the 
focal-point assessment was not given until the 
second half of the semester and that that many 
of these students withdrew from the course pri- 
or to administration of the Focus Intervention. 

Case Study 

The following case study and qualitative analysis 
is presented to illustrate the specifically math- 
ematical changes that the Focus Intervention 
causes in students and to support the positive 
findings of this study. Th e student in this case 
study had failed or withdrawn from five lower- 
division mathematics classes within the past 4 
years (one attempt at prealgebra, one attempt 
at geometry, and three attempts at number 
theory). On two occasions she reported that a 
previous instructor informed her that she would 
never “get math” and she should “stop trying 
and do something else.” 

She began the treatment class with high mo- 
tivation but a significant lack of ability. It was 
not clear from her early classroom assessments 
that shed learned or retained any content in her 
previous prealgebra class coursework. Prior to 
completing the Focus Intervention, shed taken 
seven chapter tests, and her scores on each of 
these were passing but below average. Her work 
on homework and tests was hard to follow, a 
jumble of scratch work and multiple approaches. 
She displayed particular difficulty with signed 
numbers, basic algebraic manipulation, and any 
multistep problem. Figure 3 illustrates two ex- 
amples of the students typical work before the 
intervention and were both taken from the first 
attempt at the focal-point test. Although many 
of the computations are done correctly, signs, 
parentheses, and equation structure disappear 
immediately as she begins solving the problem. 


Numbers appear and 
disappear as the stu- 
dent focuses on single 
steps with no coher- 
ence to the overall 
objective of the prob- 
lem. By the end of her 
solving attempt, sev- 
eral errors have been 
made. It is the opin- 
ion of the authors that 
the students limited 
proficiency in struc- 
tured writing and use 
of math vocabulary, 
symbols and communication skills prevent her 
from solving the problem correctly. 

These behaviors appear to be typical of de- 
velopmental students and were clearly present in 
other students in the class. Although many stu- 
dents can perform single mathematical tasks or 
manipulations with some accuracy, they fail in 
solving multistep problems due to their inability 
to unite individual steps into a coherent problem- 
solving strategy. Students get lost in their own 
writing and lose focus on the problems original 
objective. Using a traditional method of partial 
credit, it is possible for many such students to 
receive passing scores without being able to com- 
pletely solve any single problem. 

Figure 4 shows the students fourth attempt 
on the focal-point assessment. It demonstrates 
improvements in organization, communication, 
and language of mathematics and in working a 
multistep problem. The student is presenting her 
manipulations in sequential order and main- 
taining the overall objective of the problems, as 
opposed to breaking the equation into pieces or 
changing the meaning of the expression by mov- 
ing parentheses as in Figure 3 . When required 
to organize and communicate her solution, the 
student demonstrates an improved understand- 
ing of the distributive property, the correct 


IS st-Ahi-v+ot' 2 ^ 
v 
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meaning and use of parentheses, and combining 
like terms (clearly absent in Figure 3). 

Figure 5 illustrates two similar problems tak- 
en from the students final exam. These problems 
illustrate that the improvements seen in Figure 
4 — structure, communication, math vocabulary, 
and symbols use-were maintained after the Fo- 
cus Intervention was complete. (The final exam 
was given 2 weeks after she completed her final 
focus -intervention assessment.) 

It is the opinion of the authors that the mas- 
tery/feedback intervention was the catalyst for 
the improvements. Th e student received sub- 
stantial feedback on many occasions prior to the 
first intervention but did not make the changes 
until held accountable to a 100% mastery crite- 
ria reference with a well-defined rubric. 

This case study shows the specifics regard- 
ing why the method is efficacious. By focusing 
on specific problems and specific rubric-based 
criteria, the instructor had a basis for addressing 
shortcomings in a criterion -referenced way, as 
opposed to saying “this is wrong.” 

Discussion 

The importance of structured and focused feed- 
back and mastery for students was supported by 
this pilot study. Student motivation and interest 
in the course was hard to quantify, but there were 
differences between the treatment and control 
groups noted in the data. Less formally, the in- 
structor (2nd author) noted course participation 
in and outside of lecture, and interest in course 
content appeared higher among the treatment 
group than among the control group at the end 
of the semester. After receiving repeated per- 
sonalized feedback focused on their reasoning 
skills, communication, and use of correct math 
notation, treatment students appeared to exhibit 
positive changes in attitudes and mathematical 
abilities. We pose that this positive change was 
due to the Focus Intervention as it provided 
students with a content focus and an opportu- 
nity to receive high-quality feedback on content 


Table 5 

Correlation Matrix for Individual Measures 


A 

B 

C 

D 

E 

B,D,E 

MSES2 

Final 

A 

1.000 








B 

.577** 

1.000 







C 

.550** 

.254 

1.000 






D 

.800** 

.692** 

.554** 

1.000 





E 

.457** 

.766** 

.375* 

.751** 

1.000 




B,D,E 

.683** 

.865** 

.462** 

.921** 

.929** 

1.000 



MSES2 

.307 

.240 

.384* 

.288 

.131 

.241 

1.000 


Final 

.238 

.229 

.136 

.091 

-.071 

.067 

.489** 

1.000 


N= 36 * ± .329 critical value .05 (two-tail) ** ± .424 critical value .01 (two-tail) 
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Figure 3. Two examples of student problem solving before 
the Focus Intervention. 
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Figure 4. Example of case-study student's problem solving 
on the fourth attempt on the focal-point assessment. 
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Figure 5. Except from case-study student's final exam. 


topics in a one-on-one, structured environment. 
In that sense, the Focus Intervention likely in- 
creased students’ understanding of how to ef- 
fectively use faculty office hours as a feedback 
component. 

It should be noted that the only statistically 
significant gain came from course-specific mea- 
sures. Although the mean gains of the treatment 
group were larger (or decreased less) than those 
of the control group in all measures, nonmath- 
ematics-specific measures were not shifted in a 
significant way. Th is fi nding suggests that dif- 
ferences in gains on nonmathematics-specific 
measures could have been due to chance and 
that generic academic attitudes and beliefs may 
be difficult to change in a single specific course. 

Limitations 

Several limitations of the study exist. First, this 
intervention was designed around a specific 
course at a specific institution. Generalizations 
of the findings should take into account the tar- 
geted populations learning needs and the spe- 
cific course objectives. 

Second, the validity checks involving fac- 
tor analysis of the study’s instruments lack the 
population size necessary for robust statistics. 
A strength is that the GAB and MSES2 survey 
items were extracted or modeled from existing 
valid and reliable instruments; however, caution 


should be used when 
applying the instru- 
ments to different 
populations. 

Third, the fi nal 
exam did not go 
through the full psy- 
chometric analysis 
necessary for com- 
plete validity and reli- 
ability assurances. A 
strength was the final 
exam’s face validity 
and that it exhibited 
some convergent va- 
lidity to the MSES2 
gains. 

Implications 
for Practice 
and Future 
Research 

It is the opinion of the 
instructors involved 
in this study that 
the Focus Interven- 
tion would be easy 
to modify and imple- 
ment into nearly any traditional mathematics 
course with little or no changes to the curricu- 
lum. The positive results from this study suggest 
that traditional classrooms could benefit greatly 
from implementation with little or no inconve- 
nience to the instructor. 

To implement the Focus Intervention into 
a mathematics course different from the one 
discussed in this article, the instructors at an 
institution should meet and collectively agree 
on which learning objectives are deemed high- 
priority for that particular course. Th is is an 
institution-based decision since college math- 
ematics courses are often preparatory for subse- 
quent courses, and specific content needs may 
differ among institutions. Instructors should 
also identify which assessment criteria are im- 
portant. Some instructors place more emphasis 
on problem-solving approaches, whereas others 
value how students represent, communicate, 
and explain their solutions. Because assessment 
criteria communicate expectations to students, 
conversations about assessment criteria should 
be connected to conversations about which 
mathematics habits are important for student 
success. Th is process can send clear messages 
to students about how to learn mathematics ef- 
fectively. 

Once the high-priority topics and assessment 
criteria are agreed upon, five parallel versions of 


the focal-point assessment and a corresponding 
scoring rubric should be created. Efforts should 
be made to ensure the assessments are parallel. 
Faculty should agree that the assessments ap- 
pear to assess the same material, which increas- 
es a type of validity of the assessments, known 
as face validity. Validity can also be increased 
by piloting the assessments with a sample of 
students. This would help determine if the as- 
sessments actually assess what is intended and 
if items assumed to be parallel are actually an- 
swered in a similar way by students. A strong, 
well -structured feedback component should be 
made available after each assessment. To ame- 
liorate the faculty time commitment necessary 
to provide such feedback, tutors, graduate assis- 
tants, Supplemental Instruction leaders, and ac- 
ademic support staff can be enlisted and trained 
to provide such feedback. 

The Focus Intervention also could be adapt- 
ed to distance education settings by making the 
assessment available in a testing center or on- 
line, if a mathematical typesetting environment 
is available. Otherwise, the assessments could 
be printed and mailed. Because a strong and 
thorough feedback component appears to be vi- 
tal to the success of the Focus Intervention, an 
interactive environment such as a chat room or 
phone conversations should be made available. 

Given the positive results observed in this 
pilot study, future research is warranted. Pilot 
studies like this offer evidence that the effort, 
time, and expense needed to run a large-scale 
study with a rigorous research design are justi- 
fied. Future research could also focus on longitu- 
dinal tracking of Focus Intervention students to 
answer questions of whether students exposed 
to the Focus Intervention maintain achievement 
gains in subsequent courses and whether they 
exhibit and maintain changes in their study hab- 
its, attitudes about mathematics learning, use 
of feedback, and use of instructional resources 
such as faculty office hours. Although this study 
lacks the experimental research design and sam- 
ple size needed to generalize the fi ndings, the 
study does offer the classroom teacher a model, 
including measures for examining the effects of 
the Focus Intervention in his or her own class- 
room. In more general terms, this study might 
encourage instructors to engage in small-scale 
studies in their own classrooms. Such research 
is often termed action research. It involves in- 
structors indentifying weaknesses in their class- 
rooms, reviewing the literature for appropriate 
interventions and measures, implementing in- 
terventions, and reflecting on the effects of in- 
terventions using qualitative and quantitative 
means. Such processes allow instructors to judi- 
ciously implement ideas from published educa- 
tion research into their classroom practices. 
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Conclusion 

Developmental algebra curriculums are of- 
ten too dense with topics (Baker, 2009) which 
are presented at such a rapid pace that many 
topics are not completely understood or mas- 
tered by students (Steinfort, 1996). Instructors 
who feel bound by established curricula may 
have difficulty narrowing their curriculum. The 
positive findings of the Focus Intervention pilot 
study suggest that instructors can select a small 
subset of high-priority learning objectives and 
expect complete mastery on those outcomes 
without sacrificing established curriculum. The 
study also suggests that some of the positive as- 
pects associated with mastery learning, includ- 
ing a strong feedback component, may also be 
obtained, though there was evidence that the 
negative effect associated with mastery learn- 
ing-higher dropout rates-may also occur with 
such an approach. Th e Focus Intervention of- 
fers an alternative approach toward the goal of 
assisting students to better understand and in- 
ternalize mathematics necessary for course and 
program success. 
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Appendix A 
Focal Point Test 

Pre-Algebra Focal Point Test Version 1 

Remember that this is an all or nothing test 
(100 points or o points). I am testing for mas- 
tery; therefore, no partial credit. Work will be 
assessed according to the scoring rubric given 
in class. Assessment includes: math communi- 
cation, use of notation, clarity of explanations, 
solution strategy chosen, and correctness. 

1. Simplify. 

(-7)2 + 2 [ -32 - ( 3 - 11 ) ] 

2. Solve the equation. Answer with an in- 
teger or a simplified fraction. 

-8x - 3( 5 - x ) = 1 + 2 (3X - 5) 

3. Evaluate and simplify. 

(-9z)(-6x + yz) for x = -2, y = 3, and z = -4 

4. Clearly identify the meaning of your 
variable, write an equation that repre- 
sents the problem, solve the equation, 
and clearly state the solution in English. 

A high school graduating class is made 
up of 435 students. Th ere are 101 more 
girls than boys. How many boys are in the 
class? 

5. Consider the items labeled A and B be- 
low and answer all three parts to this 
question (labeled i., ii, and iii.). 

i. Notice that A and B have the same 
LCD. What is it? 

ii. In your own words, explain at least 
two differences between A and B. 

iii. Find an appropriate answer to both 
A and B showing all work. 


Appendix B 

Rubric for FPT Feedback 

Scoring Rubric 

To receive the score on the left, your work must 
demonstrate the characteristics on the right. 

4 Advanced application of basic skills. 

3 Barely proficient application of basic skills. 

2 Development toward proficient application 
of basic skills. 

1 Minimal development of basic skills. 
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